{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End of the distribution imputation\n",
    "\n",
    "On occasions, one has reasons to suspect that missing values are not missing at random. And if the value is missing, there has to be a reason for it. Therefore, we would like to capture this information.\n",
    "\n",
    "Adding an additional variable indicating missingness may help with this task (as we discussed in the previous lecture). However, the values are still missing in the original variable, and they need to be replaced if we plan to use the variable in machine learning.\n",
    "\n",
    "Sometimes, we may also not want to increase the feature space by adding a variable to capture missingness.\n",
    "\n",
    "So what can we do instead?\n",
    "\n",
    "We can replace the NA, by values that are at the far end of the distribution of the variable.\n",
    "\n",
    "The rationale is that if the value is missing, it has to be for a reason, therefore, we would not like to replace missing values for the mean and make that observation look like the majority of our observations. Instead, we want to flag that observation as different, and therefore we assign a value that is at the tail of the distribution, where observations are rarely represented in the population.\n",
    "\n",
    "### Advantages\n",
    "\n",
    "- Easy to implement\n",
    "- Captures the importance of missingess if there is one\n",
    "\n",
    "### Disadvantages\n",
    "\n",
    "- Distorts the original distribution of the variable\n",
    "- If missingess is not important, it may mask the predictive power of the original variable by distorting its distribution\n",
    "- If the number of NA is big, it will mask true outliers in the distribution\n",
    "- If the number of NA  is small, the replaced NA may be considered an outlier and pre-processed in a subsequent step of feature engineering\n",
    "\n",
    "\n",
    "### Final note\n",
    "\n",
    "I haven't seen this method used in data competitions, however, this method is used in finance companies. When capturing the financial history of customers, if some of the variables are missing, the company does not like to assume that missingness is random. Therefore, a different treatment is provided to replace them, by placing them at the end of the distribution.\n",
    "\n",
    "See my talk at pydata London for an example of feature engineering in Finance:\n",
    "https://www.youtube.com/watch?v=KHGGlozsRtA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "===============================================================================\n",
    "\n",
    "## Real Life example: \n",
    "\n",
    "### Predicting Survival on the Titanic: understanding society behaviour and beliefs\n",
    "\n",
    "Perhaps one of the most infamous shipwrecks in history, the Titanic sank after colliding with an iceberg, killing 1502 out of 2224 people on board. Interestingly, by analysing the probability of survival based on few attributes like gender, age, and social status, we can make very accurate predictions on which passengers would survive. Some groups of people were more likely to survive than others, such as women, children, and the upper-class. Therefore, we can learn about the society priorities and privileges at the time.\n",
    "\n",
    "\n",
    "### Predicting Sale Price of Houses\n",
    "\n",
    "The problem at hand aims to predict the final sale price of homes based on different explanatory variables describing aspects of residential homes. Predicting house prices is useful to identify fruitful investments, or to determine whether the price advertised for a house is over or underestimated, before making a buying judgment.\n",
    "\n",
    "=============================================================================\n",
    "\n",
    "In the following cells, I will show how this procedure impacts features and machine learning using the Titanic and House Price datasets from Kaggle.\n",
    "\n",
    "If you haven't downloaded the datasets yet, in the lecture \"Guide to setting up your computer\" in section 1, you can find the details on how to do so."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "% matplotlib inline\n",
    "\n",
    "# for regression problems\n",
    "from sklearn.linear_model import LinearRegression, Ridge\n",
    "\n",
    "# for classification\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "# to split and standarize the datasets\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# to evaluate regression models\n",
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "# to evaluate classification models\n",
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Age</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>7.2500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>38.0</td>\n",
       "      <td>71.2833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>26.0</td>\n",
       "      <td>7.9250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>35.0</td>\n",
       "      <td>53.1000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>8.0500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Survived   Age     Fare\n",
       "0         0  22.0   7.2500\n",
       "1         1  38.0  71.2833\n",
       "2         1  26.0   7.9250\n",
       "3         1  35.0  53.1000\n",
       "4         0  35.0   8.0500"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load the Titanic Dataset with a few variables for demonstration\n",
    "\n",
    "data = pd.read_csv('titanic.csv', usecols = ['Age', 'Fare','Survived'])\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Survived    0.000000\n",
       "Age         0.198653\n",
       "Fare        0.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's look at the percentage of NA\n",
    "data.isnull().mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imputation important\n",
    "\n",
    "Imputation has to be done over the training set, and then propagated to the test set. This means that when replacing by a value at the far end of the distribution, it has to be the distribution of the variable in the training set the one that we will use to replace NA both in train and test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((623, 3), (268, 3))"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's separate into training and testing set\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(data, data.Survived, test_size=0.3,\n",
    "                                                    random_state=0)\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0xc9423e2278>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW8AAAD3CAYAAADSftWOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAD1NJREFUeJzt3X2MZfVdx/H3sMND1wx0jLfEJqTE1H7/8A802xQsAhsK\nFmpwlah/GLBlo6SGRFAaoNtdGxualITdGmhalWW6WCVpgG4JJCubFORBUxsemkiKX1psYxNtMq6z\nduhay3bHP+5ZuWznPpx775l7fsv7lWxy7zn3nvOZO2c+89szv3Pv3NraGpKkspwy6wCSpPosb0kq\nkOUtSQWyvCWpQJa3JBVofiN2sry8WntKy+LiZlZWjjQRZyJtzQXtzdbWXNDebOaqr63ZJsnV6SzM\n9VvX2pH3/PymWUdYV1tzQXuztTUXtDebuepra7amcrW2vCVJ/VneklQgy1uSCmR5S1KBLG9JKpDl\nLUkFsrwlqUCWtyQVyPKWpAJtyOXxap/tn3p83eVLt106k+1IqseRtyQVyPKWpAJZ3pJUIMtbkgpk\neUtSgSxvSSqQ5S1JBbK8JalAlrckFcjylqQCWd6SVCDLW5IKZHlLUoEsb0kqkOUtSQUa+n7eEbEJ\nuAcIYA34MPBDYF91/0Xghsw81lxMSVKvUUbeVwFk5oXATuCTwB5gZ2ZeBMwB2xpLKEn6CUPLOzO/\nDFxf3X0HcBjYAjxZLTsAXNZIOknSukb6GLTMPBoR9wG/CfwWcHlmrlWrV4GzBj1/cXEz8/Obaofr\ndBZqP2cjtDUXTJ5tWl/bids5mV+zppirvrZmayLXyJ9hmZkfjIhbgX8C3tKzaoHuaLyvlZUjtYN1\nOgssL6/Wfl7T2poLppNtWl9b73ZO9tesCeaqr63ZJsk1qPSHnjaJiGsj4qPV3SPAMeDZiNhaLbsS\neHqsZJKksYwy8v4S8PmIeAo4FbgJeAm4JyJOq24/2FxESdKJhpZ3Zv4A+J11Vl0y/TiSpFF4kY4k\nFWjkP1iq3bZ/6vF1ly/ddukGJ5G0ERx5S1KBLG9JKpDlLUkFsrwlqUCWtyQVyPKWpAJZ3pJUIMtb\nkgpkeUtSgSxvSSqQ5S1JBfK9TVQs389Fb2aOvCWpQJa3JBXI8pakAlneklQgy1uSCmR5S1KBnCqo\nN3D6nVQGR96SVCDLW5IKNPC0SUScCiwB5wKnA7cD3wUeBb5ZPexzmfnFBjNKkk4w7Jz3NcChzLw2\nIn4a+DrwCWBPZu5uPJ0kaV3DyvsB4MHq9hxwFNgCRERsozv6vikzV5uLKEk60cDyzsxXASJigW6J\n76R7+mRvZj4XER8DPg58ZNB2Fhc3Mz+/qXa4Tmeh9nM2wkbkuurmh9dd/sjubbW2M62sdbdz4uMn\nydHvtRh139N+/EYxV31tzdZErqFTBSPiHGA/8NnMvD8i3pqZh6vV+4G7h21jZeVI7WCdzgLLy+0b\n0M86V919TyvrJPvd6Neszr5m/f3sx1z1tTXbJLkGlf7A2SYRcTZwELg1M5eqxY9FxHuq2+8Dnhsr\nlSRpbMNG3juARWBXROyqlv0J8OmIeA34HnB9g/kkSesYds77RuDGdVZd2EwcSdIovEhHkgpkeUtS\ngSxvSSqQ5S1JBbK8JalAlrckFcjylqQC+Uk62lB+Uo80HY68JalAlrckFcjylqQCWd6SVCDLW5IK\nZHlLUoGcKqhG9JsSKGk6HHlLUoEsb0kqkOUtSQWyvCWpQJa3JBXI8pakAlneklQgy1uSCmR5S1KB\nBl5hGRGnAkvAucDpwO3AN4B9wBrwInBDZh5rNKUk6Q2GjbyvAQ5l5kXAFcBngD3AzmrZHLCt2YiS\npBMNe2+TB4AHq9tzwFFgC/BktewA8KvA/kEbWVzczPz8ptrhOp2F2s/ZCLPMVXff08ra9Nc8ze3P\n6jWaNnPV19ZsTeQaWN6Z+SpARCzQLfGdwJ2ZuVY9ZBU4a9hOVlaO1A7W6SywvLxa+3lNm3Wuuvue\nVtamv+Zpbr/Otmb9/ezHXPW1NdskuQaV/tA/WEbEOcATwBcy836g9/z2AnB4rFSSpLENLO+IOBs4\nCNyamUvV4hciYmt1+0rg6ebiSZLWM+yc9w5gEdgVEbuqZTcCd0XEacBLvH5OXJK0QYad876Rblmf\n6JJm4kiSRuFFOpJUID8GTW8a/T6abem2Szc4iTQ5R96SVCDLW5IKZHlLUoEsb0kqkOUtSQUqYraJ\nswQk6Y0ceUtSgSxvSSqQ5S1JBbK8JalAlrckFaiI2SYaX7+ZOpLK5shbkgpkeUtSgSxvSSqQ5S1J\nBbK8JalAzjaRGuZ786gJjrwlqUCWtyQVyPKWpAKNdM47Is4H7sjMrRHxS8CjwDer1Z/LzC82FVCS\n9JOGlndE3AJcC/ygWrQF2JOZu5sMJknqb5TTJq8AV/fc3wL8WkQ8FRH3RsRCM9EkSf0MHXln5kMR\ncW7Poq8BezPzuYj4GPBx4CODtrG4uJn5+U21w3U6g38vDFvflGnt96qbH679nFm90VTTr/U0t193\nW/0e3+/788jubbUz1dnvqOtnpa25oL3Zmsg1zjzv/Zl5+Pht4O5hT1hZOVJ7J53OAsvLqwMfM2x9\nE0bJdTJq+mue5vbrbqvpx4+znbYeZ23NBe3NNkmuQaU/zmyTxyLiPdXt9wHPjRNKkjS+cUbefwjc\nHRGvAd8Drp9uJEnSMCOVd2Z+B7iguv08cGGDmSRJQ3iRjiQVyDem0kj8ODWpXRx5S1KBLG9JKpDl\nLUkFsrwlqUCWtyQVyNkmUk1+rJnawJG3JBXI8pakAlneklQgy1uSCmR5S1KBTsrZJs4GkHSyc+Qt\nSQWyvCWpQJa3JBXI8pakAlneklQgy1uSCnRSThWUSuZUV43CkbckFcjylqQCjXTaJCLOB+7IzK0R\n8U5gH7AGvAjckJnHmosoSTrR0JF3RNwC7AXOqBbtAXZm5kXAHLCtuXiSpPWMctrkFeDqnvtbgCer\n2weAy6YdSpI02NDTJpn5UESc27NoLjPXqturwFnDtrG4uJn5+U21w3U6CxOtn/TxTW9Hr+s3w2Ic\npRwXs8o5qbbkWE9bszWRa5ypgr3ntxeAw8OesLJypPZOOp0FlpdXBz5m2PpJH7+eUXJptmZxXGzE\nfttw3LX5+G9rtklyDSr9cWabvBARW6vbVwJPj7ENSdIExhl53wzcExGnAS8BD043kiRpmJHKOzO/\nA1xQ3X4ZuKTBTJKkIbxIR5IK5Hub6KQzzZkrUls58pakAlneklQgy1uSCmR5S1KBLG9JKpDlLUkF\nsrwlqUCWtyQVyPKWpAJZ3pJUIMtbkgpkeUtSgSxvSSqQ5S1JBbK8JalAlrckFcjylqQCWd6SVCA/\nBm2Afh+ntXTbpRucRE3yY9Ne5zFfDkfeklQgy1uSCmR5S1KBxj7nHRHPA9+v7n47M6+bTiRJ0jBj\nlXdEnAHMZebW6caRJI1i3JH3ecDmiDhYbWNHZn6134MXFzczP7+p9k46nYWJ1k/6+Ka3o5NL08fj\noFkxj+zetu7yq25+uNbj++nN2nt7Wtuflrb+bDaRa9zyPgLcCewFfh44EBGRmUfXe/DKypHaO+h0\nFlheXh34mGHrJ31809vRyWVWx+NG7Pv440f5uRxn+9MwaraNNkmuQaU/bnm/DHwrM9eAlyPiEPCz\nwHfH3J4kqYZxZ5tsB3YDRMTbgTOB/5hWKEnSYOOOvO8F9kXEM8AasL3fKRNJ0vSNVd6Z+SPgd6ec\nRZI0Ii/SkaQCFf3GVNN6QyHfmEil8xh+83HkLUkFsrwlqUCWtyQVyPKWpAJZ3pJUoKJnm8xK3Y+K\ncibAm0Pd77PHhSbhyFuSCmR5S1KBLG9JKpDlLUkFsrwlqUBza2trje9keXm19k56P33Cv8pLJ7d+\nM7XqON4ZdWeDNW3CT9KZ67fOkbckFcjylqQCWd6SVCDLW5IKZHlLUoEsb0kqkG9MJWnmZvlmb9Oa\nWthvO4/s3lY70ygceUtSgSxvSSrQWKdNIuIU4LPAecD/Ar+fmd+aZjBJUn/jjrx/AzgjM38ZuA3Y\nPb1IkqRhxi3vXwH+DiAzvwq8e2qJJElDjfXGVBGxF3goMw9U9/8N+LnMPDrlfJKkdYw78v4+sNC7\nHYtbkjbOuOX9D8AHACLiAuCfp5ZIkjTUuBfp7Acuj4h/BOaA66YXSZI0zIZ8GIMkabq8SEeSCmR5\nS1KBLG9JKlDr3lWwjZfeR8T5wB2ZuTUi3gnsA9aAF4EbMvPYDDKdCiwB5wKnA7cD35h1tojYBNwD\nRJXjw8APZ52rJ9/bgOeAy4GjLcr1PN0puADfBj7ZhmwR8VHg14HT6P5cPtmSXB8CPlTdPQP4RboX\nD/75LLNVP5f30f25/DHwBzR0nLVx5N2qS+8j4hZgL90DBGAPsDMzL6I706aZ93sc7hrgUJXjCuAz\nLcl2FUBmXgjspFtCbch1/AfrL4H/qRa1JdcZwFxmbq3+XdeGbBGxFXgvcCFwCXBOG3IBZOa+468X\n3V/GfwT8aQuyfQCYz8z3Ap+gweO/jeXdtkvvXwGu7rm/he7oA+AAcNmGJ+p6ANhV3Z6j+9t95tky\n88vA9dXddwCH25CrcifwF8C/V/fbkus8YHNEHIyIx6trJ9qQ7f10r+HYDzwCPNqSXP8vIt4N/EJm\n/hXtyPYyMF+dQTgTeK2pXG0s7zOB/+65/+OImNnpncx8iO434Li5zDw+v3IVOGvjU0FmvpqZqxGx\nADxId5TblmxHI+I+4G7gb9uQq/pv9nJmPtazeOa5Kkfo/mJ5P93TTK14zYCfoTt4+u2eXKe0IFev\nHcCfVbfb8Jq9SveUyb/QPX14V1O52ljebb/0vvdc1QLdkeVMRMQ5wBPAFzLzflqULTM/CLyL7gH8\nlp5Vs8q1ne6FZX9P9/zoXwNva0Eu6I7W/iYz1zLzZeAQcHbP+lllOwQ8lpk/ysyk+7eL3uKZ9fH/\nViAy84lqURuO/z+m+5q9i+7/qO6j+/eCqedqY3m3/dL7F6pzgQBXAk/PIkREnA0cBG7NzKW2ZIuI\na6s/ckF3RHkMeHbWuTLz4sy8pDpH+nXg94ADs85V2U71t52IeDvd/30ebEG2Z4ArImKuyvVTwFda\nkOu4i4Gv9Nyf+fEPrPD6mYP/Ak5tKlfrZpvQ/kvvbwbuiYjTgJfonrKYhR3AIrArIo6f+74RuGvG\n2b4EfD4inqJ74N5UZWnDa3aitnwv7wX2RcQzdGckbAf+c9bZMvPRiLgY+Brdgd4NdGfCtOE1g+6M\npn/tud+G7+engaWIeJruiHsH8GwTubw8XpIK1MbTJpKkISxvSSqQ5S1JBbK8JalAlrckFcjylqQC\nWd6SVKD/Azg1nLCBR4n7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xc947d24b38>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_train.Age.hist(bins=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "73.43632005918366"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# far end of the distribution\n",
    "X_train.Age.mean()+3*X_train.Age.std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0xc946c3e160>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAAEFCAYAAABAVTQtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADHZJREFUeJzt3XtsnWUdwPFv13awG1LmgChE/gAeRAygLFwGjBgZOiMz\nEDERpmzxQkIEhDhxASMGjQ5vAeMNrYwBIsplSEAaAbkqDoUogT0IIiQSQu0q7NJxaesf79txCGy9\nnfN7X8b3k5DstPR5fzk9/fY572nftg0PDyNJijGl6gEk6a3E6EpSIKMrSYGMriQFMrqSFKhjW+/s\n7V0/oR9t6OqaTn//polN1ELONX51nc25xqeuc0F9Z5vMXHPmzGrb2vtastPt6GhvxbKT5lzjV9fZ\nnGt86joX1He2Vs3l6QVJCmR0JSmQ0ZWkQEZXkgIZXUkKZHQlKZDRlaRARleSAhldSQpkdCUpkNGV\npEBGV5ICGV1JCmR0JSmQ0ZWkQEZXkgIZXUkKZHQlKZDRlaRA2/zDlIqxbNkynnuuN/y4GzduBGDG\njBlb/X/a26cwODgUNdIWXV27sHz518KPK7Wa0a2Bvr4++vr6aOucFnrc4Zc3A/Di4Fb/cGklhl8e\nqHoEqWWMbk20dU5j5t7Hhx5zw+M3AoQfdzQjc0nbI8/pSlIgoytJgYyuJAUyupIUyOhKUiCjK0mB\njK4kBTK6khTI6EpSIKMrSYGMriQFMrqSFMjoSlIgoytJgYyuJAUyupIUyOhKUiCjK0mBjK4kBTK6\nkhTI6EpSIKMrSYGMriQFMrqSFMjoSlIgoytJgYyuJAUyupIUyOhKUiCjK0mBjK4kBTK6khTI6EpS\nIKMrSYGMriQFMrqSFMjoSlIgoytJgYyuJAUyupIUyOhKUqCWRLe7u5trrrmyFUtLCnbNNVf69dxE\nLYnuvffey5o197diaUnB1qy536/nJvL0giQFMrqSFMjoSlIgoytJgYyuJAUyupIUyOhKUiCjK0mB\njK4kBTK6khTI6EpSIKMrSYGMriQFMrqSFMjoSlIgoytJgYyuJAUyupIUyOhKUiCjK0mBjK4kBTK6\nkhTI6EpSIKMrSYGMriQFMrqSFMjoSlIgoytJgYyuJAUyupIUyOhKUiCjK0mBjK4kBTK6khTI6EpS\nIKMrSYGMriQFMrqSFKij6gEkvXmsWtUNwOLFSwFYu/YRAPbbb/8JrdfTczMvvLCOAw54H08//W8A\nFixYOKaPneyxR1v72Wens/vuezV9baMraczuvPN24NXorl59LTDx8K1efR0vvriZJ554kqeffgoY\ne3Qne+zR1u7sbOfss5c3fW1PL0gak1WruhkaGmJoaIhVq7pZu/YRcn6UnB/dsuscj56emxkY2MTQ\n0BA5P8rAwCYGBjbR03PzqB872WOPZe2HH3646WtDi3a6GzZsYPPmzXzpS2e0YvkJa2+fwuDgUNVj\nvE5//zqG/f63xfDgS/T3b/vxU9fP5fY4V3//OqZO3WHLLheKHe8zz/xny+3Vq68d945z9errtvr2\n0Xa7I7vciR67qrXBna4khWrJTnfmzJlMmzadiy66uBXLT9icObPo7V1f9Rivc+65Z9Hbv6HqMWqj\nrX0qXTtt+/FT18/l9jjXyDOOww+fxx13/AGA+fM/wNy5h7FixYUALFp04rjXXbToBK6++oo3fPvo\nH3vipI5d1drgC2mSxmjx4qWveyEtpXcDE3sxa8GChVteSNtnnzSuF9L222//SR17LGt3dra35EU6\noytpzObP/8Brbk92J7ho0Qmv+5GxsX9s83ehjWvvvPP0lqxtdCWN2cgOd8Rkd4ILFizccupjvGu1\nYhfauHarThX5QpokBTK6khTI6EpSIKMrSYGMriQFMrqSFMjoSlIgoytJgYyuJAUyupIUyOhKUiCj\nK0mBjK4kBTK6khTI6EpSIKMrSYGMriQFMrqSFMjoSlIgoytJgYyuJAUyupIUyOhKUiCjK0mBjK4k\nBTK6khTI6EpSIKMrSYGMriQFMrqSFMjoSlIgoytJgYyuJAUyupIUyOhKUiCjK0mBjK4kBTK6khTI\n6EpSoI5WLDpv3jwGBl5qxdKSgs2de2jVI2xXWhLdpUuX0tu7vhVLSwp20kknVz3CdsXTC5IUyOhK\nUiCjK0mBjK4kBTK6khTI6EpSIKMrSYGMriQFMrqSFMjoSlIgoytJgYyuJAUyupIUyOhKUiCjK0mB\njK4kBTK6khTI6EpSIKMrSYGMriQFMrqSFMjoSlIgoytJgYyuJAUyupIUyOhKUiCjK0mBjK4kBTK6\nkhTI6EpSIKMrSYGMriQFMrqSFMjoSlIgoytJgYyuJAUyupIUyOhKUiCjK0mBjK4kBeqoegAVhl8e\nYMPjN4YfEwg/7miKuaZXPYbUEka3BmbPns3g4FD4cTduHAZgxoytB669fUoFs02nq2uX4GNKMYxu\nDaxYsYLe3vVVj/GG5syZVdvZpDcjz+lKUiCjK0mBjK4kBTK6khTI6EpSIKMrSYGMriQFMrqSFMjo\nSlIgoytJgYyuJAUyupIUyOhKUiCjK0mBjK4kBTK6khTI6EpSIKMrSYGMriQFMrqSFKhteHi46hkk\n6S3Dna4kBTK6khTI6EpSIKMrSYGMriQFMrqSFMjoSlKgjmYtlFKaAvwIOBB4EfhMzvnxZq0/USml\nQ4Fv55yPSSntDVwGDAMPA6fnnIeC5+kEuoG9gB2AC4FHajBXO3ApkMo5TgM2Vz1Xw3y7An8FjgVe\nqdFcfwNeKG8+CXyjDrOllL4CHA9Mpfi6vLPquVJKpwKnljd3BA4CjgR+UOVc5WydwEqKr8tB4LO0\n6HHWzJ3ux4Adc86HA+cC323i2hOSUloG/JziEwzwPeC8nPNRQBuwqIKxTgH6yhk+BPywJnN9FCDn\nPA84jyIedZhr5Avip8BA+aa6zLUj0JZzPqb8b0kdZkspHQMcAcwD5gN71mGunPNlI/cVxTfQM4Cv\nVj1XaSHQkXM+Avg6LXz8NzO6RwK/B8g5/xk4pIlrT9QTwAkNt99P8R0f4Bbgg+ETwW+A88t/t1F8\nN618rpzzDcDnypvvAv5Xh7lK3wF+AjxT3q7LXAcC01NKPSml21NKh9VktuOAfwDXA78DbqrJXACk\nlA4B3pNz/lmN5noM6Cifse8EvNyq2ZoZ3Z2A5xtuD6aUmnb6YiJyztdS3Hkj2nLOI7/3vB54WwUz\nbcg5r08pzQJ+S7GrrHyucrZXUkorgUuAK+swV/mUtDfnfGvDmyufq7SJ4hvCcRSnY2pxnwFvp9j0\nfLxhrik1mGvEcuCC8t91uL8ANlCcWlhLcZrtYlo0WzOj+wIwq3HtnPMrTVy/GRrPx8yi2M2FSynt\nCdwBrMo5X1WXuQByzp8G9qV44E1reFdVcy0Fjk0p/ZHiHODlwK41mAuK3dEVOefhnPNjQB+wW8P7\nq5qtD7g15/xSzjlTnJtvDEaVj/2dgZRzvqN8U10e+1+kuM/2pXgGs5LifPiIps3WzOjeS3FehPJp\n1j+auHazPFie7wL4MHB39AAppd2AHuDLOefuGs21uHzxBYod3BDwQNVz5ZyPzjnPL88DPgR8Cril\n6rlKSylfu0gpvYPi2V5PDWa7B/hQSqmtnGsGcFsN5gI4Grit4Xblj/1SP68+U18HdNKi2Zr59P96\nih3JfRTnKpc0ce1mOQe4NKU0FXiU4ul9tOVAF3B+Smnk3O6ZwMUVz3Ud8MuU0l0UD7izylmqvr/e\nSB0+jwC/AC5LKd1D8Qr3UuC/Vc+Wc74ppXQ08BeKjdXpFD9ZUYf7LAH/arhdl8/l94HulNLdFDvc\n5cADrZjNSztKUiB/OUKSAhldSQpkdCUpkNGVpEBGV5ICGV3VWkrpgJTScErpxKpnkZrB6KrullD8\nfORpVQ8iNYM/p6vaKq/d8R/gKOA+4NCc8xPlbwldQnGxoD8B+zdcuvPHwGyK36r7Qs75wUqGl7bC\nna7q7CPAU+V1DW4APl9e5nEVcHLO+WBee0GjlcCynPP7KK6YdnX0wNJojK7qbAnwq/Lfv6a4APbB\nwHM557+Xb+8GSCnNBOZS/CrzQ8BVwMyU0uzQiaVRVHrpRWlryr8UsRA4JKV0JsX1PLooLjzyRpuF\ndmBzzvmghjX2oLh4iVQb7nRVV6cAt+Wc98g575VzfhfF1fyPA7pSSu8t/79PAsM55+eBf6aUTgFI\nKR0L3FXF4NK2uNNVXS2huNJTox8By4AFwOUppSEg8+qf8TkZ+En5Z5peAj7RcBFqqRb86QW9qZR/\nTuVbwAU5540ppbOBd+acz6l4NGlMPL2gN5Xyr7GuA9aUL5gdDXyz2qmksXOnK0mB3OlKUiCjK0mB\njK4kBTK6khTI6EpSoP8DuLyGhF8aeZAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xc9423c3dd8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# we see that there are a few outliers for Age, according to its distribution\n",
    "# these outliers will be masked when we replace NA by values at the far end \n",
    "# see below\n",
    "\n",
    "sns.boxplot('Age', data=data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def impute_na(df, variable, median, extreme):\n",
    "    df[variable+'_far_end'] = df[variable].fillna(extreme)\n",
    "    df[variable].fillna(median, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Age</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Age_far_end</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>857</th>\n",
       "      <td>1</td>\n",
       "      <td>51.0</td>\n",
       "      <td>26.5500</td>\n",
       "      <td>51.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>1</td>\n",
       "      <td>49.0</td>\n",
       "      <td>76.7292</td>\n",
       "      <td>49.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>386</th>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>46.9000</td>\n",
       "      <td>1.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124</th>\n",
       "      <td>0</td>\n",
       "      <td>54.0</td>\n",
       "      <td>77.2875</td>\n",
       "      <td>54.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>578</th>\n",
       "      <td>0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>14.4583</td>\n",
       "      <td>73.43632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>549</th>\n",
       "      <td>1</td>\n",
       "      <td>8.0</td>\n",
       "      <td>36.7500</td>\n",
       "      <td>8.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>247.5208</td>\n",
       "      <td>24.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>20.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>157</th>\n",
       "      <td>0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>30.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>1</td>\n",
       "      <td>24.0</td>\n",
       "      <td>7.1417</td>\n",
       "      <td>24.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>653</th>\n",
       "      <td>1</td>\n",
       "      <td>29.0</td>\n",
       "      <td>7.8292</td>\n",
       "      <td>73.43632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>235</th>\n",
       "      <td>0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>7.5500</td>\n",
       "      <td>73.43632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>785</th>\n",
       "      <td>0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>25.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>241</th>\n",
       "      <td>1</td>\n",
       "      <td>29.0</td>\n",
       "      <td>15.5000</td>\n",
       "      <td>73.43632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>351</th>\n",
       "      <td>0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>35.0000</td>\n",
       "      <td>73.43632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>862</th>\n",
       "      <td>1</td>\n",
       "      <td>48.0</td>\n",
       "      <td>25.9292</td>\n",
       "      <td>48.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>851</th>\n",
       "      <td>0</td>\n",
       "      <td>74.0</td>\n",
       "      <td>7.7750</td>\n",
       "      <td>74.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>753</th>\n",
       "      <td>0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>7.8958</td>\n",
       "      <td>23.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>532</th>\n",
       "      <td>0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>7.2292</td>\n",
       "      <td>17.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>485</th>\n",
       "      <td>0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>25.4667</td>\n",
       "      <td>73.43632</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Survived   Age      Fare  Age_far_end\n",
       "857         1  51.0   26.5500     51.00000\n",
       "52          1  49.0   76.7292     49.00000\n",
       "386         0   1.0   46.9000      1.00000\n",
       "124         0  54.0   77.2875     54.00000\n",
       "578         0  29.0   14.4583     73.43632\n",
       "549         1   8.0   36.7500      8.00000\n",
       "118         0  24.0  247.5208     24.00000\n",
       "12          0  20.0    8.0500     20.00000\n",
       "157         0  30.0    8.0500     30.00000\n",
       "127         1  24.0    7.1417     24.00000\n",
       "653         1  29.0    7.8292     73.43632\n",
       "235         0  29.0    7.5500     73.43632\n",
       "785         0  25.0    7.2500     25.00000\n",
       "241         1  29.0   15.5000     73.43632\n",
       "351         0  29.0   35.0000     73.43632\n",
       "862         1  48.0   25.9292     48.00000\n",
       "851         0  74.0    7.7750     74.00000\n",
       "753         0  23.0    7.8958     23.00000\n",
       "532         0  17.0    7.2292     17.00000\n",
       "485         0  29.0   25.4667     73.43632"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's replace the NA with the median value in the training and testing sets\n",
    "impute_na(X_train, 'Age', X_train.Age.median(), X_train.Age.mean()+3*X_train.Age.std())\n",
    "impute_na(X_test, 'Age', X_train.Age.median(), X_train.Age.mean()+3*X_train.Age.std())\n",
    "\n",
    "X_train.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0xc943a9ffd0>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAD5CAYAAADY+KXfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEg5JREFUeJzt3X+s3XV9x/HnbS9tV3PorvOgMRJZ4nyHmaALBhgdbWdK\nAOdkIdmPbMqQjbmkDnUkKFC2zIDKoujq4nBFVnQzi4IsQtbBBsIKzhAcJHbim8BkZnEud3hbLqut\nlN79cb43njXtPed7ftzzvZ8+H0mT8/2e8/2cV29PX+dzP+d7zplaWFhAklSGVZMOIEkaHUtdkgpi\nqUtSQSx1SSqIpS5JBbHUJakg0/3cKCLOBm7KzC0RcQqwE5gBVgOXZuYzEXEF8G7gMHBDZt4zrtCS\npGOb6nWeekRcDbwT+N/MPCcidgF/n5lfjIhfBNYD3wD+EXgzsA54GHhzZh5aauzZ2fnaJ8nPzKxn\nbu5A3cPGzlz1NTVbU3NBc7OZq75hsrXbranjXdfP8sszwCVd2xuB10TEPwG/BTwInAU8kpmHMnM/\n8DRwxkBpe5ieXj2OYYdmrvqamq2puaC52cxV37iy9Sz1zLwTeLFr12nAXGZuBb4LfAA4GdjfdZt5\nYMPoYkqS+tHXmvpRngO+Ul2+G7gReAxodd2mBezrNdDMzPqBnq3a7VbvG02Auepraram5oLmZjNX\nfePINkipPwy8Ffg8sAn4N+BR4MaIWAesBU4H9vYaaJD1pHa7xezsfO3jxs1c9TU1W1NzQXOzmau+\nYbIt9WQwyCmNVwGXRsTXgAuBD2fm94EdwB7gAeC6zDw4wNiSpCH0NVPPzGeBc6rL/wGcf4zb7KRz\nqqMkaUJ885EkFcRSl6SCWOqSVBBLXZIKMsgpjVJPl3/0gWPuv+2Db1nmJNKJxZm6JBXEUpekgljq\nklQQS12SCmKpS1JBLHVJKoilLkkFsdQlqSCWuiQVxFKXpIJY6pJUEEtdkgpiqUtSQSx1SSpIX6Ue\nEWdHxINH7fvNiPiXru0rIuKxiPh6RLxtxDklSX3oWeoRcTVwK7Cua9/PAb8DTFXbrwKuBDYCFwAf\niYi14wgsSTq+fmbqzwCXLG5ExE8BHwbe13Wbs4BHMvNQZu4HngbOGGVQSVJvPUs9M+8EXgSIiNXA\nZ4E/BOa7bnYysL9rex7YMLqYkqR+1P06uzOBnwH+gs5yzM9GxCeBB4BW1+1awL5eg83MrGd6enXN\nCNBut3rfaALM1dvRWZqUrVtTc0Fzs5mrvnFkq1Xqmfko8AaAiDgN+NvMfF+1pn5jRKwD1gKnA3t7\njTc3d6B24Ha7xezsfO8bLjNz9ac7S9OyLWpqLmhuNnPVN0y2pZ4MRnJKY2Z+H9gB7KEza78uMw+O\nYmxJUv/6mqln5rPAOUvty8ydwM4RZpMk1eSbjySpIJa6JBXEUpekgljqklQQS12SCmKpS1JBLHVJ\nKoilLkkFsdQlqSCWuiQVxFKXpIJY6pJUEEtdkgpiqUtSQSx1SSqIpS5JBbHUJakglrokFcRSl6SC\n9PUdpRFxNnBTZm6JiDcBnwJeAg4Bl2bmf0fEFcC7gcPADZl5z7hCS5KOredMPSKuBm4F1lW7/gz4\ng8zcAnwZ+EBEvAq4EtgIXAB8JCLWjiWxJOm4+ll+eQa4pGv7NzLzieryNHAQOAt4JDMPZeZ+4Gng\njJEmlST11HP5JTPvjIjTurb/CyAizgXeA2yiMzvf33XYPLCh19gzM+uZnl5dMzK0263axywHc/V2\ndJYmZevW1FzQ3Gzmqm8c2fpaUz9aRPw6cB3wS5k5GxHPA93pWsC+XuPMzR2ofd/tdovZ2fnax42b\nufrTnaVp2RY1NRc0N5u56hsm21JPBrVLPSLeQecF0S2Z+YNq96PAjRGxDlgLnA7srR9VkjSMWqUe\nEauBHcB3gS9HBMBDmfnHEbED2ENnnf66zDw46rCSpKX1VeqZ+SxwTrX58uPcZiewczSxJEmD8M1H\nklQQS12SCmKpS1JBLHVJKoilLkkFsdQlqSCWuiQVxFKXpIJY6pJUEEtdkgpiqUtSQSx1SSqIpS5J\nBbHUJakglrokFcRSl6SCWOqSVBBLXZIKYqlLUkH6+o7SiDgbuCkzt0TE64BdwAKwF9iWmUci4grg\n3cBh4IbMvGdMmSVJx9Fzph4RVwO3AuuqXTcD2zPzPGAKuDgiXgVcCWwELgA+EhFrxxNZknQ8/Sy/\nPANc0rV9JvBQdXk3sBU4C3gkMw9l5n7gaeCMUQaVJPXWc/klM++MiNO6dk1l5kJ1eR7YAJwM7O+6\nzeL+Jc3MrGd6enX/aSvtdqv2McvBXL0dnaVJ2bo1NRc0N5u56htHtr7W1I9ypOtyC9gHPF9dPnr/\nkubmDtS+83a7xezsfO3jxs1c/enO0rRsi5qaC5qbzVz1DZNtqSeDQc5+eTwitlSXLwL2AI8C50XE\nuojYAJxO50VUSdIyGmSmfhWwMyLWAE8Cd2TmSxGxg07BrwKuy8yDI8wpSepDX6Wemc8C51SXnwI2\nH+M2O4GdowwnSarHNx9JUkEsdUkqiKUuSQWx1CWpIJa6JBXEUpekgljqklQQS12SCmKpS1JBLHVJ\nKoilLkkFsdQlqSCWuiQVxFKXpIJY6pJUEEtdkgpiqUtSQSx1SSqIpS5JBRnki6eJiJOA24HTgJeA\nK4DDwC5gAdgLbMvMIyNJKUnqy6Az9bcC05l5LvAh4EbgZmB7Zp4HTAEXjyaiJKlfg5b6U8B0RKwC\nTgZeBM4EHqqu3w1sHT6eJKmOgZZfgBfoLL18G3gF8DZgU2YuVNfPAxt6DTIzs57p6dW177zdbtU+\nZjmYq7ejszQpW7em5oLmZjNXfePINmipvx+4NzOviYhTgQeANV3Xt4B9vQaZmztQ+47b7Razs/O1\njxs3c/WnO0vTsi1qai5objZz1TdMtqWeDAZdfpkD9leXfwCcBDweEVuqfRcBewYcW5I0oEFn6p8A\nbouIPXRm6NcCjwE7I2IN8CRwx2giSpL6NVCpZ+YLwK8d46rNw8WRJA3DNx9JUkEsdUkqiKUuSQWx\n1CWpIJa6JBXEUpekgljqklQQS12SCmKpS1JBLHVJKoilLkkFsdQlqSCWuiQVxFKXpIJY6pJUEEtd\nkgpiqUtSQSx1SSqIpS5JBRn0i6eJiGuAt9P54ulPAw8Bu4AFYC+wLTOPjCCjJKlPA83UI2ILcC6w\nkc6XTZ8K3Axsz8zzgCng4hFllCT1adDllwuAbwJ3AXcD9wBn0pmtA+wGtg6dTpJUy6DLL68AXgu8\nDfhp4CvAqsxcqK6fBzb0GmRmZj3T06tr33m73ap9zHIwV29HZ2lStm5NzQXNzWau+saRbdBSfw74\ndmb+CMiIOEhnCWZRC9jXa5C5uQO177jdbjE7O1/7uHEzV3+6szQt26Km5oLmZjNXfcNkW+rJYNDl\nl4eBCyNiKiJeDbwMuL9aawe4CNgz4NiSpAENNFPPzHsiYhPwKJ0nhm3Ad4CdEbEGeBK4Y2QpJUl9\nGfiUxsy8+hi7Nw+RRZI0JN98JEkFsdQlqSCWuiQVxFKXpIJY6pJUEEtdkgpiqUtSQSx1SSqIpS5J\nBbHUJakglrokFcRSl6SCWOqSVBBLXZIKYqlLUkEsdUkqiKUuSQWx1CWpIJa6JBVk4O8oBYiIU4Bv\nAOcDh4FdwAKwF9iWmUeGDShJ6t/AM/WIOAn4DPDDatfNwPbMPA+YAi4ePp4kqY5hll8+BtwCfK/a\nPhN4qLq8G9g6xNiSpAEMtPwSEZcBs5l5b0RcU+2eysyF6vI8sKHXODMz65meXl37/tvtVu1jloO5\nejs6S5OydWtqLmhuNnPVN45sg66pXw4sRMRW4E3A54BTuq5vAft6DTI3d6D2HbfbLWZn52sfN27m\n6k93lqZlW9TUXNDcbOaqb5hsSz0ZDLT8kpmbMnNzZm4BngAuBXZHxJbqJhcBewYZW5I0uKHOfjnK\nVcDOiFgDPAncMcKxJUl9GLrUq9n6os3DjidJGpxvPpKkgljqklQQS12SCmKpS1JBRnn2iwpw+Ucf\nOOb+2z74lmVOImkQztQlqSCWuiQVxFKXpIJY6pJUEEtdkgri2S8F80wW6cTjTF2SCuJMXUXxtxOd\n6JypS1JBnKmrL86ApZXBUl9BLNYfO97PQjrRufwiSQWx1CWpIAMtv0TEScBtwGnAWuAG4FvALmAB\n2Atsy8wjI0mp4rm0JI3GoGvq7wCey8x3RsTLgSeqP9sz88GIuAW4GLhrRDlVCNfCpfEadPnlS8D1\n1eUp4DBwJvBQtW83sHW4aJKkugaaqWfmCwAR0QLuALYDH8vMheom88CGkSSUJPVt4FMaI+JUOssr\nn87ML0TEn3Zd3QL29RpjZmY909Ora993u92qfcxymFSuuvc7ypzj/juPavxJ/oxGranZzFXfOLIN\n+kLpK4H7gPdk5v3V7scjYktmPghcBHy11zhzcwdq33e73WJ2dr72ceM2yVx173eUOcf9dx7V+Mcb\nZ6W9QOvjv56m5oLhsi31ZDDoTP1aYAa4PiIW19bfC+yIiDXAk3SWZSRJy2jQNfX30inxo20eLk7/\nVtoMS5KWgx8TcALytEKpXJa6NCH+tqlx8GMCJKkgztQbqO7yiMspy8sZtprMUtdQfEKRmsVSl0bE\nGbyawDV1SSrICTNTX2qZwJmUpFKcMKUurXQu76gflroazRdipXpcU5ekgjhT1wnBGb9OFM7UJakg\nlrokFcRSl6SCuKY+AE8tK59r8D/mezxWFktdWuHqTjKWY1LixGdyiiv1Uc6wnK1pJfPxe2JyTV2S\nCjLSmXpErAI+DbwROAT8bmY+Pcr7aLJR/Rqssvj5+FpOUwsLCyMbLCIuAd6emZdFxDnANZl58fFu\nPzs7X/vO2+0Ws7PzPvClE8Cwa/C9+mKSa/yL2QY8dup41416Tf0XgH8AyMyvR8SbRzy+pBPIpH77\nHeWTwPHGuvvjx53vDmXUa+onA/u7tl+KiOJejJWkphr18svNwNcz84vV9n9m5mtGdgeSpCWNeqb+\nCPBWgGpN/ZsjHl+StIRRL43cBZwfEV8DpoB3jXh8SdISRrr8IkmaLN98JEkFsdQlqSCWuiQVZEWc\nQ97Ujx+IiLOBmzJzS0S8DtgFLAB7gW2ZeWSZ85wE3AacBqwFbgC+NelcVbbVwE4gqiy/DxxsQrYq\n3ynAN4DzgcNNyBUR/wo8X21+B7ixCbmqbNcAbwfW0Pm/+dCks0XEZcBl1eY64E103hD5yQnnOgm4\nnc7/y5eAKxjjY2ylzNR/BViXmT8PfBD4+ITzEBFXA7fSefAA3Axsz8zz6Jz5M563iy3tHcBzVYYL\ngT9vSC6AXwbIzI3AdjoF1Yhs1X+6zwA/rHZNPFdErAOmMnNL9eddTchVZdsCnAtsBDYDpzYhW2bu\nWvx50XmCvhL4o0nnonOa93Rmngt8iDE/9ldKqf+/jx8AmvDxA88Al3Rtn0lntgKwG9i67IngS8D1\n1eUpOrOBJuQiM/8O+L1q87XAPhqSDfgYcAvwvWq7CbneCKyPiPsi4oHqfR9NyAVwAZ33oNwF3A3c\nQ3OyUX08yRsy8y8bkuspYLpacTgZeHGcuVZKqTfu4wcy8046/ziLpjJz8fzQeWDDBDK9kJnzEdEC\n7qAzI554rq58hyPiduBTwN80IVv1K/tsZt7btXviuYADdJ5sLqCzVNWIn1flFXQmVr/alW1VQ7IB\nXAv8SXW5CT+zF+gsvXybzhLkjnHmWiml/jzQ6tpelZmHJxXmOLrXw1p0ZqLLLiJOBb4KfD4zv9CU\nXIsy87eB19N5cP9E11WTynY5nTfMPUhnDfZzwCkNyPUU8NeZuZCZTwHPAa9sQC6qLPdm5o8yM+m8\nNtJdSpN8/P8kEJn51WpXEx7/76fz83o9nd/AbqfzWsRYcq2UUl8JHz/weLXWCHARsGe5A0TEK4H7\ngA9k5m1NyVVle2f14hp0ZqFHgMcmnS0zN2Xm5mod9gngUmD3pHPRebL5OEBEvJrOb6v3NSAXwMPA\nhRExVWV7GXB/Q7JtAu7v2m7C43+OH680/AA4aZy5VsTZL6yMjx+4CtgZEWuAJ+ksfyy3a4EZ4PqI\nWFxbfy+wY8K5AL4M/FVE/DOdB/X7qjyT/pkdSxP+LT8L7IqIh+mcIXE58D8NyEVm3hMRm4BH6UwM\nt9E5O2fi2eicXfXvXdtN+Lf8BHBbROyhM0O/FnhsXLn8mABJKshKWX6RJPXBUpekgljqklQQS12S\nCmKpS1JBLHVJKoilLkkFsdQlqSD/B5ODpWiUupgQAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xc947d24a20>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# we see an accumulation of values around the median for the median imputation\n",
    "X_train.Age.hist(bins=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0xc948440cc0>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAD3CAYAAADi8sSvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAD7hJREFUeJzt3W2MHeV5xvH/4rVxHS3uVhyoIqHQKsqtqFKCZBQo1C+K\nTIE0DRVSXz4QSlApldxCIyTeYlo1AjVE4LROhEhNXZO0+RITqmLJhQpS11BVlBSkWDE3gibKh0bV\nFtaw1IHEePthxmWF7N2dOW+zj/8/CemcOTsz1zk+e51nnzMzTMzPzyNJKsMZ4w4gSRocS12SCmKp\nS1JBLHVJKoilLkkFmRznzmdm5hofejM9vY7Z2aPDiNMXczXX1WxdzQXdzWau5vrJ1utNTZzqsRU3\nUp+cXDXuCCdlrua6mq2ruaC72czV3LCyrbhSlySdmqUuSQWx1CWpIJa6JBXEUpekgljqklQQS12S\nCmKpS1JBlnVGaURcBNybmVsi4gLgy8A7wNvAtZn53xFxA3AjcAy4OzP3DSu0JOnkliz1iLgV+DTw\nv/WivwT+KDNfiIgbgdsi4ovATcCFwFrg6Yj4p8x8e0i5Jalv13/hqZMu3337x0ecZHCWM1J/Bbga\n+Hp9/3cy80cL1n8L+BjwTF3ib0fEy8BHgH9fbMPT0+tanSrb6001XmcUzNVcV7N1NRd0N1tJuUb1\nXIaxnyVLPTMfiYjzF9z/EUBEXAL8IbAJuBx4fcFqc8D6pbbd5mI2vd4UMzNzjdcbNnM119VsXc0F\n3c1WWq5RPJd+XrPFPgxafVEaEb8NPAj8WmbOAG8AC/cyBRxps21JUnuNL70bEddQfSG6JTNfqxc/\nC9wTEWuBM4EPA4cGllKStCyNSj0iVgE7gR8C34oIgAOZ+acRsRM4SDX6/1xmvjXosJKkxS2r1DPz\nB8DF9d2fO8XP7AJ2DSaWJKkNTz6SpIJY6pJUEEtdkgpiqUtSQSx1SSqIpS5JBbHUJakglrokFcRS\nl6SCWOqSVBBLXZIKYqlLUkEsdUkqiKUuSQWx1CWpIJa6JBXEUpekgljqklQQS12SCmKpS1JBLHVJ\nKoilLkkFsdQlqSCWuiQVxFKXpIJY6pJUkMnl/FBEXATcm5lbIuKDwB5gHjgEbMvM4xFxA3AjcAy4\nOzP3DSmzJOkUlhypR8StwEPA2nrRDmB7Zm4EJoCrIuLngZuAS4HLgT+PiDOHE1mSdCrLmX55Bbh6\nwf0NwIH69n5gK/Ax4JnMfDszXwdeBj4yyKCSpKUtOf2SmY9ExPkLFk1k5nx9ew5YD5wFvL7gZ04s\nX9T09DomJ1ctP22t15tqvM4omKu5rmbrai7obraSco3quQxjP8uaU3+P4wtuTwFHgDfq2+9dvqjZ\n2aONd97rTTEzM9d4vWEzV3NdzdbVXNDdbKXlGsVz6ec1W+zDoM3RL89HxJb69pXAQeBZYGNErI2I\n9cCHqb5ElSSNUJuR+i3ArohYAxwG9mbmOxGxk6rgzwA+l5lvDTCnJGkZllXqmfkD4OL69kvA5pP8\nzC5g1yDDSZKa8eQjSSqIpS5JBbHUJakglrokFcRSl6SCWOqSVBBLXZIKYqlLUkEsdUkqiKUuSQWx\n1CWpIJa6JBXEUpekgljqklQQS12SCmKpS1JBLHVJKoilLkkFsdQlqSCWuiQVxFKXpIJY6pJUEEtd\nkgpiqUtSQSx1SSqIpS5JBZlss1JErAYeBs4H3gFuAI4Be4B54BCwLTOPDySlJGlZ2o7UPwFMZuYl\nwOeBe4AdwPbM3AhMAFcNJqIkabnalvpLwGREnAGcBfwU2AAcqB/fD2ztP54kqYlW0y/Am1RTLy8C\nZwOfBDZl5nz9+BywfqmNTE+vY3JyVeOd93pTjdcZBXM119VsXc0F3c1WUq5RPZdh7KdtqX8WeDwz\n74iI84CngDULHp8Cjiy1kdnZo4133OtNMTMz13i9YTNXc13N1tVc0N1speUaxXPp5zVb7MOg7fTL\nLPB6ffs1YDXwfERsqZddCRxsuW1JUkttR+pfAnZHxEGqEfqdwHPArohYAxwG9g4moiRpuVqVema+\nCfzWSR7a3F8cSVI/PPlIkgpiqUtSQSx1SSqIpS5JBbHUJakglrokFcRSl6SCWOqSVBBLXZIKYqlL\nUkEsdUkqiKUuSQWx1CWpIJa6JBXEUpekgljqklQQS12SCmKpS1JBLHVJKoilLkkFsdQlqSCWuiQV\nxFKXpIJY6pJUEEtdkgpiqUtSQSbbrhgRdwCfAtYADwAHgD3APHAI2JaZxweQUZK0TK1G6hGxBbgE\nuBTYDJwH7AC2Z+ZGYAK4akAZJUnL1Hb65XLgu8CjwGPAPmAD1WgdYD+wte90kqRG2k6/nA18APgk\n8AvAPwBnZOZ8/fgcsH6pjUxPr2NyclXjnfd6U43XGQVzNdfVbF3NBd3NVlKuUT2XYeynbam/CryY\nmT8BMiLeopqCOWEKOLLURmZnjzbeca83xczMXOP1hs1czXU1W1dzQXezlZZrFM+ln9dssQ+DttMv\nTwNXRMRERLwfeB/wZD3XDnAlcLDltiVJLbUaqWfmvojYBDxL9cGwDfg+sCsi1gCHgb0DSylJWpbW\nhzRm5q0nWby5jyySpD558pEkFcRSl6SCWOqSVBBLXZIKYqlLUkEsdUkqiKUuSQWx1CWpIJa6JBXE\nUpekgljqklQQS12SCmKpS1JBLHVJKoilLkkFsdQlqSCWuiQVxFKXpIJY6pJUEEtdkgpiqUtSQSx1\nSSqIpS5JBbHUJakglrokFcRSl6SCWOqSVJDJflaOiHOA7wCXAceAPcA8cAjYlpnH+w0oSVq+1iP1\niFgNfBX4cb1oB7A9MzcCE8BV/ceTJDXRz0j9PuBB4I76/gbgQH17P/CrwKOLbWB6eh2Tk6sa77jX\nm2q8ziiYq7muZutqLuhutpJyjeq5DGM/rUo9Iq4DZjLz8Yg4UeoTmTlf354D1i+1ndnZo4333etN\nMTMz13i9YTNXc13N1tVc0N1speUaxXPp5zVb7MOg7Uj9emA+IrYCFwBfA85Z8PgUcKTltiVJLbWa\nU8/MTZm5OTO3AC8A1wL7I2JL/SNXAgcHklCStGx9Hf3yHrcAuyJiDXAY2DvAbUuSlqHvUq9H6yds\n7nd7kqT2PPlIkgpiqUtSQSx1SSqIpS5JBbHUJakglrokFWSQx6mrANd/4amTLt99+8fHsh1JzThS\nl6SCWOqSVBBLXZIKYqlLUkEsdUkqiKUuSQXxkMaCeVihdPpxpC5JBbHUJakglrokFcQ5dRXF7xF0\nurPUtSyWpbQyOP0iSQVxpL6COFp+16leC+l050hdkgpiqUtSQZx+USc4tSQNhiN1SSqII3WNlF9w\nSsPVqtQjYjWwGzgfOBO4G/gesAeYBw4B2zLz+EBSSpKWpe30yzXAq5m5EbgC+AqwA9heL5sArhpM\nREnScrUt9W8Cd9W3J4BjwAbgQL1sP7C1v2iSpKZaTb9k5psAETEF7AW2A/dl5nz9I3PA+qW2Mz29\njsnJVY333+tNNV5nFMaVq+l+B5lz2M95UNsf52s0aF3NVlKuUT2XYeyn9RelEXEe8CjwQGZ+IyK+\nuODhKeDIUtuYnT3aeL+93hQzM3ON1xu2ceZqut9B5hz2cx7U9k+1nZV2KKXv/2ba5hrFc+nnNVvs\nw6DV9EtEnAs8AdyWmbvrxc9HxJb69pXAwTbbliS113akficwDdwVESfm1m8GdkbEGuAw1bTM0Ky0\nEZYkjULbOfWbqUr8vTb3F0eS1A9PPjoNeQKQVC5LXRoTpxA1DF77RZIK4ki9g5pOjzidMlqOsNVl\njtQlqSCO1NUX/0qQusWRuiQVxJG6NCDOtasLTptSX2yawF86SaVw+kWSCnLajNSllc7pHS2HI3VJ\nKogjdXWah0xKzVjqOi344aDThdMvklQQS12SCmKpS1JBLHVJKoilLkkF8eiXFjwJpHweLfMuL7Gx\nsljq0grXdJAxikGJA5/xcfpFkgpS3Eh9kH82+ye4VjLfv6cnR+qSVJDiRurjNKi5TZXF/5G4Rmli\nfn5+bDufmZlrvPNeb4qZmTnf+NJpoN8vVtv2xSi+0D2RreW6E6d6bKAj9Yg4A3gA+CjwNvB7mfny\nIPchSTq1QU+//AawNjN/OSIuBu4HrhrwPiSdJsY1pTnIQzJPta3H7h9ONQ76i9JfAf4RIDP/Dbhw\nwNuXJC1ioHPqEfEQ8Ehm7q/v/xD4xcw8NrCdSJJOadAj9TeAqYXbt9AlaXQGXerPAJ8AqOfUvzvg\n7UuSFjHoL0ofBS6LiH8FJoDPDHj7kqRFjPU4dUnSYHmZAEkqiKUuSQWx1CWpICvigl5dvfxARFwE\n3JuZWyLig8AeYB44BGzLzOMjzrMa2A2cD5wJ3A18b9y56myrgF1A1Fn+AHirC9nqfOcA3wEuA451\nIVdE/AfVYcIA3wfu6UKuOtsdwKeANVS/mwfGnS0irgOuq++uBS6gOiHyL8acazXwMNXv5TvADQzx\nPbZSRur/f/kB4Haqyw+MVUTcCjxE9eYB2AFsz8yNVEf+jOPyCNcAr9YZrgC+0pFcAL8OkJmXAtup\nCqoT2epfuq8CP64XjT1XRKwFJjJzS/3fZ7qQq862BbgEuBTYDJzXhWyZuefE60X1AX0T8CfjzkV1\nmPdkZl4CfJ4hv/dXSql38fIDrwBXL7i/gWq0ArAf2DryRPBN4K769gTVaKALucjMvwd+v777AeAI\nHckG3Ac8CPxXfb8LuT4KrIuIJyLiqfq8jy7kAric6hyUR4HHgH10JxsRcSHwS5n5Vx3J9RIwWc84\nnAX8dJi5VkqpnwW8vuD+OxEx1qmjzHyE6h/nhInMPHF86BywfgyZ3szMuYiYAvZSjYjHnmtBvmMR\n8TDwZeDvupCt/pN9JjMfX7B47LmAo1QfNpdTTVV14vWqnU01sPrNBdnO6Eg2gDuBP6tvd+E1e5Nq\n6uVFqinIncPMtVJKfSVcfmDhfNgU1Uh05CLiPODbwNcz8xtdyXVCZv4u8CGqN/fPLHhoXNmupzph\n7p+p5mC/BpzTgVwvAX+bmfOZ+RLwKnBuB3JRZ3k8M3+SmUn13cjCUhrn+/9ngcjMb9eLuvD+/yzV\n6/Uhqr/AHqb6LmIouVZKqa+Eyw88X881AlwJHBx1gIg4F3gCuC0zd3clV53t0/WXa1CNQo8Dz407\nW2ZuyszN9TzsC8C1wP5x56L6sLkfICLeT/XX6hMdyAXwNHBFREzU2d4HPNmRbJuAJxfc78L7f5Z3\nZxpeA1YPM9eKOPqFlXH5gVuAXRGxBjhMNf0xancC08BdEXFibv1mYOeYcwF8C/ibiPgXqjf1H9d5\nxv2anUwX/i3/GtgTEU9THSFxPfA/HchFZu6LiE3As1QDw21UR+eMPRvV0VX/ueB+F/4tvwTsjoiD\nVCP0O4HnhpXLywRIUkFWyvSLJGkZLHVJKoilLkkFsdQlqSCWuiQVxFKXpIJY6pJUkP8DGbnE9OFS\nsZoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xc948260ac8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# we see an accumulation of values at the far end for the far end imputation\n",
    "\n",
    "X_train.Age_far_end.hist(bins=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0xc948682a58>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAAEFCAYAAABAVTQtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADNhJREFUeJzt3X2QXXdZwPHvZjcxyTaFbUlxsAjOQJ9RcWhrY6TQl3Ho\nFKst6kCYIVFJbZUZpIBMUDKgwVF8qQhTnMqLlhYsSFFgkBGptiC0aEmxHeiITykqM8ofLMlK0+1L\n0s31j9/Z6Vay7W5y73NP7Pcz05ncvZvfeXr37veee3LvuRODwQBJUo014x5Akp5IjK4kFTK6klTI\n6EpSIaMrSYWmHuvK2dkDq35pw8zMRubm7j/6iUakr3NBf2dzrtXr62zOtXrHMtvmzZsmlrtu6Hu6\nU1OTw15yKPo6F/R3Nudavb7O5lyrN6rZPLwgSYWMriQVMrqSVMjoSlIhoytJhYyuJBUyupJUyOhK\nUiGjK0mFjK4kFTK6klTI6EpSIaMrSYWMriQVMrqSVMjoSlIhoytJhYyuJBUyupJU6DE/mFL98ta3\n7mFubn/Z9iYn17CwcPhxv29+fh6A6enpUY8ErHyucVjJbDMzJ7F7956agdQ7Rvc4Mje3n3379jGx\ndsO4R3mUwaEHAXhoYdkPQFVncOiBcY+gMTO6x5mJtRs44VmXjHuMR7nvnk8A9G6uPlq8rfTE5TFd\nSSpkdCWpkNGVpEJGV5IKGV1JKmR0JamQ0ZWkQkZXkgoZXUkqZHQlqZDRlaRCRleSChldSSpkdCWp\nkNGVpEJGV5IKGV1JKmR0JamQ0ZWkQkZXkgoZXUkqZHQlqZDRlaRCRleSChldSSpkdCWpkNGVpEJG\nV5IKGV1JKmR0JamQ0ZWkQkZXkgoZXUkqZHQlqZDRlaRCRleSChldSSpkdCWpkNGVpEJGV5IKGV1J\nKjQ1ikVvuOF6ALZt2z6K5SVppG644Xo2bFjHxRe/dOhrj2RPd+/e29i797ZRLC1JI7d3723ceuut\nI1nbwwuSVMjoSlIhoytJhYyuJBUyupJUyOhKUiGjK0mFjK4kFTK6klTI6EpSIaMrSYWMriQVMrqS\nVMjoSlIhoytJhYyuJBUyupJUyOhKUiGjK0mFjK4kFTK6klTI6EpSIaMrSYWMriQVMrqSVMjoSlIh\noytJhYyuJBUyupJUyOhKUiGjK0mFjK4kFTK6klTI6EpSIaMrSYWMriQVMrqSVMjoSlIhoytJhYyu\nJBUyupJUaGoUi87Pz3Pw4EPs2nXFKJY/KpOTa1hYODzuMY5opbPNze1n4OPkcW2wcJC5uQfLfzf6\nev/v61xzc/tZv379SNb2N1iSCo1kT3d6eprp6WmuvPKqUSx/VDZv3sTs7IFxj3FEK51t164r2H/v\n/QUTaVQmJtcxc+LG8t+Nvt7/+zrXrl1XMDk5mn1S93QlqZDRlaRCRleSChldSSpkdCWpkNGVpEJG\nV5IKGV1JKmR0JamQ0ZWkQkZXkgoZXUkqZHQlqZDRlaRCRleSChldSSpkdCWpkNGVpEJGV5IKGV1J\nKmR0JamQ0ZWkQkZXkgoZXUkqZHQlqZDRlaRCRleSChldSSpkdCWpkNGVpEJGV5IKGV1JKmR0JamQ\n0ZWkQkZXkgoZXUkqZHQlqZDRlaRCRleSChldSSpkdCWpkNGVpEJTo1h0y5ato1hWkkps2bKVDRvW\njWTtkUR327bto1hWkkps27adzZs3MTt7YOhre3hBkgoZXUkqZHQlqZDRlaRCRleSChldSSpkdCWp\nkNGVpEJGV5IKGV1JKmR0JamQ0ZWkQkZXkgoZXUkqZHQlqZDRlaRCRleSChldSSpkdCWpkNGVpEJG\nV5IKGV1JKmR0JamQ0ZWkQkZXkgoZXUkqZHQlqZDRlaRCRleSChldSSpkdCWpkNGVpEJGV5IKGV1J\nKmR0JamQ0ZWkQkZXkgoZXUkqZHQlqZDRlaRCU+MeQKszOPQA993ziXGP8SiDQw8A9G6uPmq31cZx\nj6ExMrrHkZmZk0q3Nzm5hoWFw4/7ffPzAwCmp2tistK5xuHxZ9tY/nNUvxjd48ju3XtKt7d58yZm\nZw+UbnMl+joX9Hs29YPHdCWpkNGVpEJGV5IKGV1JKmR0JamQ0ZWkQkZXkgoZXUkqZHQlqZDRlaRC\nRleSChldSSpkdCWpkNGVpEJGV5IKGV1JKmR0JamQ0ZWkQkZXkgoZXUkqNDEYDMY9gyQ9YbinK0mF\njK4kFTK6klTI6EpSIaMrSYWMriQVMrqSVGhqGItExBrgauC5wEPAZZl5zzDWPhYRsRX4g8w8PyKe\nBVwLDIC7gFdl5uHiedYC1wDPBL4H+B3gX8c9VzfbJPBeILpZXgk82IfZuvlOAb4EXAA83Ie5IuJf\ngHu7i/8B/G4f5upmeyNwCbCO9rv5j+OeLSJeAbyiu7geOB14AfCOcc7VzbYWuI72u7kAXM6I7mfD\n2tP9GWB9Zj4P+A3gbUNa96hFxBuAP6P9cAH+GHhTZp4DTAAvHsNYO4B93QwvAv6kJ3MBXAyQmc8H\n3kQLSC9m634h3g080H1p7HNFxHpgIjPP7/7b2Ye5utnOB84Gng+cBzy9D7Nl5rWLtxftAfQK4DfH\nPVfnImAqM88GfpsR3v+HFd0XAH8HkJn/DJw1pHWPxdeBn1ty+Udpj/YAnwJeWD4RfAR4c/fnCdoj\naR/mIjM/Dvxyd/EZwP/Qk9mAPwLeBXyzu9yHuZ4LbIyIGyPi5oj48Z7MBXAh8BXgY8DfAJ+kP7MR\nEWcBP5yZ7+nRXHcDU92z9hOBQ6OabVjRPRH4zpLLCxExlEMXRysz/5p2wy2ayMzF9zwfAJ40hpnu\ny8wDEbEJ+CvaHuXY51oy38MRcR3wTuD6PszWPSWdzcxPL/ny2OcC7qc9GFxIOxTTi9ur8xTajs9L\nl8y2piezAewG3tL9uS+32X20Qwv/RjvMdhUjmm1Y0b0X2LR03cx8eEhrD8vSYzGbaHty5SLi6cBn\ngA9k5gf7MteizPxF4DTaHW/DkqvGNdulwAUR8VnaMcD3A6f0YK67gb/IzEFm3g3sA57ag7noZvl0\nZh7MzKQdm18ajHHe/58MRGZ+pvtSX+7/r6PdZqfRnsVcRzsevmhosw0rurfSjonQPc36ypDWHaY7\numNdAD8JfL56gIh4KnAj8OuZeU1f5upm+/nuH1+g7cUdBm4f92yZeW5mntcdB7wT+AXgU+Oei/Zg\n8DaAiHga7dnejT2YC+AW4EURMdHNNg3c1JPZzgVuWnK5F/d/YI5Hnq3vB9YyotmGdQjgY7S9kS/Q\njlXuHNK6w/R64L0RsQ74Ku3pfbXdwAzw5ohYPLb7GuCqMc8F8FHgfRHxOdod7rXdPOO+zY6kDz/L\nPweujYhbaP+6fSnw7R7MRWZ+MiLOBb5I27F6Fe3VFWOfjfbqmH9fcrkPP0uAtwPXRMTnaXu4u4Hb\nRzGbp3aUpEK+OUKSChldSSpkdCWpkNGVpEJGV5IKjfVdY/r/IyKeQ3t99ku6dwMOc+0TgZtp99dt\n3ZsRxiIi9gBk5p5xzaDjm9HVsOykvY7xlcBQo0t7J9rBzOzDOT2kY+LrdHXMuvNs/DdwDvAFYGtm\nfr17N887aSf2+Sfgh5acZvNPgZNp7357dWbesczap3Rrfi9tb3cH7Y0JpwJPAz5He5faecAfApPA\nXd3bmY+03iRwJXB+973XZubbu1l3d/P8IG2v/eWZeTAidtFOBvRt2juXvuiero6Wx3Q1DD8FfKN7\n2v9x4Fe60zF+ANiemWfw6JMPXQe8ITPPpMXsL5dbODO/BVwG3J6Zl3TburM7jeizgecBZ3bffhrw\nE8sFt3N5t+6ZwI8BL46Ic7rrzgZ+lRbd7wcu7M6IdSlwBu0sU6eu4PaQlmV0NQw7gQ91f/4w7UTV\nZwDfyswvd1+/BiAiTgC20N5yfCfwQeCEiDh5JRvKzA8Bfx8Rr6XtRZ8MnPDI1fmdZf9y80Lgkm7b\nt9Ei+iPddXdl5n91J6r+KnASbY/4b7szxM3TTs8pHTWP6eqYdE//LwLOiojX0M69MUM7QciRHtQn\ngQcz8/Qla5xKO8nISrb3auAlwHuAfwCe020THjnJ+WOZpO1lf7Rb7ynAPLCVdjauRYNu3cH/+f94\nuFtDOiru6epY7QBuysxTM/OZmfkM2ln3LwRmImJxL/LlwKDbE/1aROwAiIgLaMdlV+oC4N2ZeT0t\niKezugjeDFweEWu7ve5baMFdzk3AT0fEk7pPi/jZVWxL+i5GV8dqJ+0zuJa6mhbDHcD7I+JLtI+M\nWdwT3Q5cFhFfBn4PeNmSk0U/nncAv9V9PtnVtH9k+4FVzPsu4GvAHbSzSL0vMz+73Ddn5p3dNvfS\nPkXgG6vYlvRdfPWCRqL72JPfB96SmfMR8WvA92Xm68c8mjRWHtPVSGTm4YjYD+yNiIPAfwK/tNz3\nR8TrgCO96uCbmXnRarYdES8D3nik65YeS5bGwT1dSSrkMV1JKmR0JamQ0ZWkQkZXkgoZXUkq9L/F\n42gHLXuQtgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xc94826d828>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# indeed, far end imputation now indicates that there are no outliers in the variable\n",
    "sns.boxplot('Age_far_end', data=X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0xc9486d3ba8>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAAEFCAYAAABAVTQtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADjxJREFUeJzt3X+QnPVdwPH33SWQu0siR01DoQ2ZKfabRJnSCoPSljCM\nQMWRc4jWmQDaRFGmHVOkBvQKCgqMFIpMaitKjdAUpNi0BTvUZqRIW6s1bWGESe9Lg7aZgRLP5Gx+\nEuDu/ON5bnN33CW3l93PPgnv1wzD7u2z3/3k7vZ9zz67t9c2MjKCJClGe6sHkKTXE6MrSYGMriQF\nMrqSFMjoSlKgWYe6cGBgd90vbejp6WJwcN/MJ2qSqs4F1Z3NuepX1dmcq35HMtuCBfPaprqs4Xu6\ns2Z1NHrJhqjqXFDd2ZyrflWdzbnq16zZPLwgSYGMriQFMrqSFMjoSlIgoytJgYyuJAUyupIUyOhK\nUiCjK0mBjK4kBTK6khTI6EpSIKMrSYGMriQFMrqSFMjoSlIgoytJgYyuJAUyupIU6JB/mFLVcuut\nNzI4uLMpa+/duxeA7u7u2sc6OtoZGhpuyu319JxIX9+NTVlbqjKjexQZHNzJjh07aJvd2fC1R155\nCYADQ1P+EdMG3tb+pt+GVFVG9yjTNruTuadd0vB192x9BKApa091W9Lrkcd0JSmQ0ZWkQEZXkgIZ\nXUkKZHQlKZDRlaRARleSAhldSQpkdCUpkNGVpEBGV5ICGV1JCmR0JSmQ0ZWkQEZXkgIZXUkKZHQl\nKZDRlaRARleSAhldSQpkdCUpkNGVpEBGV5ICGV1JCmR0JSmQ0ZWkQEZXkgIZXUkKZHQlKZDRlaRA\nRleSAhldSQpkdCUpkNGVpEBGV5ICGV1JCmR0JSmQ0ZWkQEZXkgIZXUkKZHQlKVBlovvQQ/fz0EP3\nt3oMqVK8Xxx7KhPdzZu/xebN32r1GFKleL849lQmupL0emB0JSmQ0ZWkQEZXkgIZXUkKZHQlKZDR\nlaRARleSAhldSQpkdCUpkNGVpEBGV5ICGV1JCmR0JSmQ0ZWkQEZXkgIZXUkKZHQlKZDRlaRARleS\nAhldSQpkdCUpkNGVpEBGV5ICGV1JCmR0JSmQ0ZWkQEZXkgIZXUkKZHQlKZDRlaRARleSAhldSQpk\ndCUpkNGVpEBGV5ICGV1JCmR0JSnQrGYs2t+/hY0bP8uPfvQ8b3rTKXzkIzcBcPXVVwFw1113s27d\nHQCsWfMHzRhBOqZs2vQoABdeePFrzk+8L0227dy5czjnnPMB2LBhPQBXXLF60u37+7cAsGTJsnGn\nD7dtvfr7t/Dii12cdNLiSdeezvVnetv1ztZITYnuww9v5Lnnvg9Q+z/Arl27aqefeuq7zbhp6Zj0\n8MOfBw4Gaez5ifelybZta6MW3See+CpwMLqv3X4jUMRs7OnDbVv/v2kjs2d3cM01fZOuPZ3rz/S2\n652tkRoe3aeffpqcvzfuY7fc8icMDGyvnb/yyitqp9etu8O9XekQ9u/fx/79+4CDe4Oj56+/fm1t\nu3Xr7mDJkmVTbrtp06Ns3/4iw8PDQLHHu3DhSeMuX7Roce3+u2nTo7XT/f1b2LbtB1Nu29+/pa74\n9fdvOeTahwvvxOs3MrzNXBugbWRkZMoLBwZ2T33hFO6881aeeeaZuq6zfv0DfOADv8XLLx+gp+fE\nem9yWjo62hkaGm7K2kdqurMNDu5kmHbmpV9t+Ax7tj4CwNzTLmn42hPtzp+jneEZf62Pha/ldA0O\n7qxFEqCzsws4GNKJOju7apdN3Lazs4sDB16qrdfe3s7xx88Zd/miRafWgjN2rZSWsm3bD6fcNqWl\nXHfdDdP+d91225+Nu+7EtT/xiU/Vdf16brve2Way9oIF89qmuswn0iQpUMMPL6xcuZK+vvHHQd76\n1p9iYGB77ZhuR0cHQ0NDAJxxxjsB6O7upru7m9tvX9fokQBYsGAeAwO7m7L2kZrubGvXrmHnrsn3\ncI4mbR3H0TO/a8Zf62Phazlda9euYf/+fezbV3zde3svBeDBBz8DwMknn8ILLzwPFPelJUuW1S6b\nuG1v76Vs3/4ijz/+zwAsX34+CxeeNO7yRYsW89GP3lw7f/CyFWzb9oNDbLuirn9Xb++KcdeduHa9\n12+kZq4NTYju6aefTkpLxx3XHX31wurVKwG4554NtdMez5UOrbOzi9GjgBOfSLv55ttfc1+a6om0\n0fOHeyItpaW1808++R2geLKqeGJt8m3rPe65ZMkyUlrK7NkdU649nevP5Lbrna3RmvLqhd7eFeNe\nMjZq/vz5tdOje7iSDm/i3t/Y8xPvS5NtO3funNr55cvPP8z2KyY9fbht69Xbu4ITTuiacu3pXL9Z\nJs7WSA1/Im2mD6/Wrl0D4OGFQxg9vNCMJ7sin0jbs/URTvTwwrQ06n5R1c9ZVeeCI5vNJ9IkqSKM\nriQFMrqSFMjoSlIgoytJgYyuJAUyupIUyOhKUiCjK0mBjK4kBTK6khTI6EpSIKMrSYGMriQFMrqS\nFMjoSlIgoytJgYyuJAUyupIUyOhKUiCjK0mBjK4kBTK6khTI6EpSIKMrSYGMriQFMrqSFMjoSlIg\noytJgYyuJAUyupIUyOhKUiCjK0mBjK4kBTK6khTI6EpSIKMrSYGMriQFMrqSFGhWqwcYddZZZ7d6\nBKlyvF8ceyoT3fe977JWjyBVjveLY4+HFyQpkNGVpEBGV5ICGV1JCmR0JSmQ0ZWkQEZXkgIZXUkK\nZHQlKZDRlaRARleSAhldSQpkdCUpkNGVpEBGV5ICGV1JCmR0JSmQ0ZWkQEZXkgIZXUkKZHQlKZDR\nlaRARleSAhldSQpkdCUpkNGVpEBGV5ICGV1JCmR0JSmQ0ZWkQEZXkgIZXUkKZHQlKZDRlaRARleS\nAhldSQpkdCUpkNGVpEBGV5ICGV1JCjSr1QOoPiOv7GfP1keasi7QlLUnv62upt+OVEVG9yjS03Ni\n09beu3cEgO7ugzHs6GhnaGi4CbfW1dR/i1RlRvco0td3Y+jtLVgwj4GB3aG3KR3rPKYrSYGMriQF\nMrqSFMjoSlIgoytJgYyuJAUyupIUyOhKUiCjK0mBjK4kBTK6khTI6EpSIKMrSYGMriQFMrqSFMjo\nSlIgoytJgYyuJAUyupIUyOhKUqC2kZGRVs8gSa8b7ulKUiCjK0mBjK4kBTK6khTI6EpSIKMrSYGM\nriQFmtWIRVJK7cAngbcDB4DfzjlvbcTaRyKldDZwW875vJTSacC9wAjwDPDBnPNw8DyzgfXAYuB4\n4GZgS6vnKmfrAO4BUjnLVcBLVZitnO+NwHeAC4BXqzBXSum7wK7y7H8Dt1RhrnK2PwIuAY6juG8+\n0erZUkrvB95fnp0DnAG8G7irlXOVs80G7qO4bw4BV9Kk77NG7en+CjAn5/zzwB8CH2vQujOWUroW\n+BTFFxfgTuD6nPN7gDagtwVjXQ7sKGd4L/CXFZkL4JcBcs7vAq6nCEglZivvEH8N7C8/1PK5Ukpz\ngLac83nlf6uqMFc523nAOcC7gOXAW6owW8753tHPF8UP0DXAH7d6rtLFwKyc8znAn9LE7/9GRffd\nwD8B5Jz/HTizQeseieeAS8ec/1mKn/YAXwZ+IXwi+AfghvJ0G8VP0irMRc75i8DvlGdPBf6PiswG\n3AHcDbxQnq/CXG8HulJKm1JKX00p/VxF5gK4CHga+ALwj8CXqM5spJTOBH465/w3FZrrWWBW+ah9\nPvBKs2ZrVHTnAz8ec34opdSQQxczlXPeSPGJG9WWcx79nefdwE+0YKY9OefdKaV5wOco9ihbPteY\n+V5NKd0HfBy4vwqzlQ9JB3LOXxnz4ZbPBeyj+GFwEcWhmEp8vko/SbHj82tjZmuvyGwAfcBN5emq\nfM72UBxa6Kc4zLaOJs3WqOjuAuaNXTfn/GqD1m6Uscdi5lHsyYVLKb0FeBzYkHN+oCpzjco5/ybw\nNopvvM4xF7VqttXABSmlf6E4Bvhp4I0VmOtZ4DM555Gc87PADmBhBeainOUrOeeXc86Z4tj82GC0\n8vv/BCDlnB8vP1SV7//fp/icvY3iUcx9FMfDRzVstkZF918pjolQPsx6ukHrNtKT5bEugF8Evh49\nQEppIbAJuC7nvL4qc5WzXVE++QLFXtww8O1Wz5ZzPjfnvLw8DvgU8BvAl1s9F8UPg48BpJROpni0\nt6kCcwF8A3hvSqmtnK0beKwis50LPDbmfCW+/4FBDj5a3wnMpkmzNeoQwBco9ka+SXGsclWD1m2k\nDwP3pJSOA75H8fA+Wh/QA9yQUho9tvshYF2L5wL4PPB3KaWvUXzDXV3O0+rP2WSq8LX8W+DelNI3\nKJ7dXg38bwXmIuf8pZTSucB/UOxYfZDi1RUtn43i1TH/NeZ8Fb6WAH8BrE8pfZ1iD7cP+HYzZvOt\nHSUpkL8cIUmBjK4kBTK6khTI6EpSIKMrSYGMriotpfQzKaWRlNKKVs8iNYLRVdWtonh95FWtHkRq\nBF+nq8oq37/jeeA9wDeBs3POz5W/JfRxijcM+jdg2Zi37/wr4A0Uv1X3eznnJ1syvDQF93RVZb8E\n/LB8b4MvAr9bvs3jBuCynPM7GP+mRvcB1+ac30nxjmkPRg8sHY7RVZWtAv6+PP1ZijfAfgfwPznn\n/yw/vh4gpTQXOIviV5mfAh4A5qaU3hA6sXQYLX37RWkq5V+KuBg4M6X0IYr39OiheOORyXYWOoCX\ncs5njFnjzRRvXiJVhnu6qqrLgcdyzm/OOS/OOZ9K8W7+FwE9KaXTy+1WAiM55x8D308pXQ6QUroA\n+ForBpcOxT1dVdUqind6GuuTwLXAhcCnU0rDQObgn/G5DLi7/FNNLwO/PuZNqKVK8NULOqqUf07l\nz4Gbcs57U0rXAKfknD/c4tGkafHwgo4q5V9j3QlsLp8wOxe4tbVTSdPnnq4kBXJPV5ICGV1JCmR0\nJSmQ0ZWkQEZXkgL9P8vDp7RiGqA7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xc9487476d8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# on the other hand, replacing values by the median, now generates the impression of a higher\n",
    "# amount of outliers\n",
    "\n",
    "sns.boxplot('Age', data=X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "scaler = StandardScaler()\n",
    "X_train = pd.DataFrame(scaler.fit_transform(X_train))\n",
    "X_test = pd.DataFrame(scaler.transform(X_test))\n",
    "\n",
    "X_train.columns = ['Survived','Age','Fare','Age_far_end']\n",
    "X_test.columns = ['Survived','Age','Fare','Age_far_end']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Logistic Regression roc-auc: 0.6794863451985858\n",
      "Test set\n",
      "Logistic Regression roc-auc: 0.7244940476190476\n",
      "Train set\n",
      "Logistic Regression roc-auc: 0.6781577406130019\n",
      "Test set\n",
      "Logistic Regression roc-auc: 0.7197619047619047\n"
     ]
    }
   ],
   "source": [
    "# we compare the models built using Age filled with median, vs Age filled with values at the far end of the distribution\n",
    "# variable indicating missingness\n",
    "\n",
    "logit = LogisticRegression(random_state=44, C=1000) # c big to avoid regularization\n",
    "logit.fit(X_train[['Age','Fare']], y_train)\n",
    "print('Train set')\n",
    "pred = logit.predict_proba(X_train[['Age','Fare']])\n",
    "print('Logistic Regression roc-auc: {}'.format(roc_auc_score(y_train, pred[:,1])))\n",
    "print('Test set')\n",
    "pred = logit.predict_proba(X_test[['Age','Fare']])\n",
    "print('Logistic Regression roc-auc: {}'.format(roc_auc_score(y_test, pred[:,1])))\n",
    "\n",
    "logit = LogisticRegression(random_state=44, C=1000) # c big to avoid regularization\n",
    "logit.fit(X_train[['Age_far_end','Fare']], y_train)\n",
    "print('Train set')\n",
    "pred = logit.predict_proba(X_train[['Age_far_end','Fare']])\n",
    "print('Logistic Regression roc-auc: {}'.format(roc_auc_score(y_train, pred[:,1])))\n",
    "print('Test set')\n",
    "pred = logit.predict_proba(X_test[['Age_far_end','Fare']])\n",
    "print('Logistic Regression roc-auc: {}'.format(roc_auc_score(y_test, pred[:,1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Support Vector Machine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "SVC roc-auc: 0.692978460337086\n",
      "Test set\n",
      "SVC roc-auc: 0.7418154761904762\n",
      "Train set\n",
      "SVC roc-auc: 0.6816175354113793\n",
      "Test set\n",
      "SVC roc-auc: 0.7411904761904762\n"
     ]
    }
   ],
   "source": [
    "# we compare the models built using Age filled with median, vs Age filled with values at the far end of the distribution\n",
    "# variable indicating missingness\n",
    "\n",
    "SVM_model = SVC(random_state=44, probability=True, max_iter=-1, kernel='linear')\n",
    "SVM_model.fit(X_train[['Age', 'Fare']], y_train)\n",
    "print('Train set')\n",
    "pred = SVM_model.predict_proba(X_train[['Age', 'Fare']])\n",
    "print('SVC roc-auc: {}'.format(roc_auc_score(y_train, pred[:,1])))\n",
    "print('Test set')\n",
    "pred = SVM_model.predict_proba(X_test[['Age', 'Fare']])\n",
    "print('SVC roc-auc: {}'.format(roc_auc_score(y_test, pred[:,1])))\n",
    "\n",
    "# model build using natural distributions\n",
    "\n",
    "SVM_model = SVC(random_state=44, probability=True, max_iter=-1, kernel='linear')\n",
    "SVM_model.fit(X_train[['Age_far_end','Fare']], y_train)\n",
    "print('Train set')\n",
    "pred = SVM_model.predict_proba(X_train[['Age_far_end','Fare']])\n",
    "print('SVC roc-auc: {}'.format(roc_auc_score(y_train, pred[:,1])))\n",
    "print('Test set')\n",
    "pred = SVM_model.predict_proba(X_test[['Age_far_end','Fare']])\n",
    "print('SVC roc-auc: {}'.format(roc_auc_score(y_test, pred[:,1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the titanic dataset, replacing missing values in Age with the median obtains better results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### House Sale Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# we are going to train a model on the following variables,\n",
    "\n",
    "cols_to_use = ['OverallQual', 'TotalBsmtSF', '1stFlrSF', 'GrLivArea','WoodDeckSF', 'BsmtUnfSF',\n",
    "               'LotFrontage', 'MasVnrArea', 'GarageYrBlt']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1460, 10)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>OverallQual</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>BsmtUnfSF</th>\n",
       "      <th>TotalBsmtSF</th>\n",
       "      <th>1stFlrSF</th>\n",
       "      <th>GrLivArea</th>\n",
       "      <th>GarageYrBlt</th>\n",
       "      <th>WoodDeckSF</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>65.0</td>\n",
       "      <td>7</td>\n",
       "      <td>196.0</td>\n",
       "      <td>150</td>\n",
       "      <td>856</td>\n",
       "      <td>856</td>\n",
       "      <td>1710</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>0</td>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>80.0</td>\n",
       "      <td>6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>284</td>\n",
       "      <td>1262</td>\n",
       "      <td>1262</td>\n",
       "      <td>1262</td>\n",
       "      <td>1976.0</td>\n",
       "      <td>298</td>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>68.0</td>\n",
       "      <td>7</td>\n",
       "      <td>162.0</td>\n",
       "      <td>434</td>\n",
       "      <td>920</td>\n",
       "      <td>920</td>\n",
       "      <td>1786</td>\n",
       "      <td>2001.0</td>\n",
       "      <td>0</td>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>60.0</td>\n",
       "      <td>7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>540</td>\n",
       "      <td>756</td>\n",
       "      <td>961</td>\n",
       "      <td>1717</td>\n",
       "      <td>1998.0</td>\n",
       "      <td>0</td>\n",
       "      <td>140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>84.0</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>490</td>\n",
       "      <td>1145</td>\n",
       "      <td>1145</td>\n",
       "      <td>2198</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>192</td>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   LotFrontage  OverallQual  MasVnrArea  BsmtUnfSF  TotalBsmtSF  1stFlrSF  \\\n",
       "0         65.0            7       196.0        150          856       856   \n",
       "1         80.0            6         0.0        284         1262      1262   \n",
       "2         68.0            7       162.0        434          920       920   \n",
       "3         60.0            7         0.0        540          756       961   \n",
       "4         84.0            8       350.0        490         1145      1145   \n",
       "\n",
       "   GrLivArea  GarageYrBlt  WoodDeckSF  SalePrice  \n",
       "0       1710       2003.0           0     208500  \n",
       "1       1262       1976.0         298     181500  \n",
       "2       1786       2001.0           0     223500  \n",
       "3       1717       1998.0           0     140000  \n",
       "4       2198       2000.0         192     250000  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's load the House Sale Price dataset\n",
    "\n",
    "data = pd.read_csv('houseprice.csv', usecols=cols_to_use+['SalePrice'])\n",
    "print(data.shape)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LotFrontage    0.177397\n",
       "OverallQual    0.000000\n",
       "MasVnrArea     0.005479\n",
       "BsmtUnfSF      0.000000\n",
       "TotalBsmtSF    0.000000\n",
       "1stFlrSF       0.000000\n",
       "GrLivArea      0.000000\n",
       "GarageYrBlt    0.055479\n",
       "WoodDeckSF     0.000000\n",
       "SalePrice      0.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's inspect the columns with missing values\n",
    "data.isnull().mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1022, 10), (438, 10))"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's separate into training and testing set\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(data, data.SalePrice, test_size=0.3,\n",
    "                                                    random_state=0)\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LotFrontage      69.0\n",
       "MasVnrArea        0.0\n",
       "GarageYrBlt    1979.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's look at the median of the variables with NA\n",
    "\n",
    "X_train[['LotFrontage', 'MasVnrArea', 'GarageYrBlt']].median()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# let's impute the NA with the median\n",
    "# remember that we need to impute with the median for the train set, and then propagate to test set\n",
    "\n",
    "median = X_train.LotFrontage.median()\n",
    "extreme = X_train.LotFrontage.mean()+3*X_train.LotFrontage.std()\n",
    "impute_na(X_train, 'LotFrontage', median, extreme)\n",
    "impute_na(X_test, 'LotFrontage', median, extreme)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "median = X_train.MasVnrArea.median()\n",
    "extreme = X_train.MasVnrArea.mean()+3*X_train.MasVnrArea.std()\n",
    "impute_na(X_train, 'MasVnrArea', median, extreme)\n",
    "impute_na(X_test, 'MasVnrArea', median, extreme)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "median = X_train.GarageYrBlt.median()\n",
    "extreme = X_train.GarageYrBlt.mean()+3*X_train.GarageYrBlt.std()\n",
    "impute_na(X_train, 'GarageYrBlt', median, extreme)\n",
    "impute_na(X_test, 'GarageYrBlt', median, extreme)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LotFrontage            0.0\n",
       "OverallQual            0.0\n",
       "MasVnrArea             0.0\n",
       "BsmtUnfSF              0.0\n",
       "TotalBsmtSF            0.0\n",
       "1stFlrSF               0.0\n",
       "GrLivArea              0.0\n",
       "GarageYrBlt            0.0\n",
       "WoodDeckSF             0.0\n",
       "SalePrice              0.0\n",
       "LotFrontage_far_end    0.0\n",
       "MasVnrArea_far_end     0.0\n",
       "GarageYrBlt_far_end    0.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.isnull().mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['LotFrontage', 'OverallQual', 'MasVnrArea', 'BsmtUnfSF', 'TotalBsmtSF',\n",
       "       '1stFlrSF', 'GrLivArea', 'GarageYrBlt', 'WoodDeckSF'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create a list with the untransformed columns\n",
    "cols_to_use_median = X_train.columns[:-4]\n",
    "cols_to_use_median"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['OverallQual',\n",
       " 'BsmtUnfSF',\n",
       " 'TotalBsmtSF',\n",
       " '1stFlrSF',\n",
       " 'GrLivArea',\n",
       " 'WoodDeckSF',\n",
       " 'LotFrontage_far_end',\n",
       " 'MasVnrArea_far_end',\n",
       " 'GarageYrBlt_far_end']"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols_to_use_extreme = list(X_train.columns)\n",
    "cols_to_use_extreme = [col for col in cols_to_use_extreme if col not in ['LotFrontage',\n",
    "                                                         'MasVnrArea',\n",
    "                                                        'GarageYrBlt',\n",
    "                                                        'SalePrice']]\n",
    "cols_to_use_extreme"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# let's standarise the dataset\n",
    "scaler = StandardScaler()\n",
    "X_train_median = scaler.fit_transform(X_train[cols_to_use_median])\n",
    "X_test_median = scaler.transform(X_test[cols_to_use_median])\n",
    "\n",
    "scaler = StandardScaler()\n",
    "X_train_extreme = scaler.fit_transform(X_train[cols_to_use_extreme])\n",
    "X_test_extreme = scaler.transform(X_test[cols_to_use_extreme])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Linear Regression mse: 1161895545.483203\n",
      "Test set\n",
      "Linear Regression mse: 2212393764.7463093\n",
      "\n",
      "Train set\n",
      "Linear Regression mse: 1216146367.5075955\n",
      "Test set\n",
      "Linear Regression mse: 2161266505.3352327\n",
      "\n"
     ]
    }
   ],
   "source": [
    "linreg = LinearRegression()\n",
    "linreg.fit(X_train_median, y_train)\n",
    "print('Train set')\n",
    "pred = linreg.predict(X_train_median)\n",
    "print('Linear Regression mse: {}'.format(mean_squared_error(y_train, pred)))\n",
    "print('Test set')\n",
    "pred = linreg.predict(X_test_median)\n",
    "print('Linear Regression mse: {}'.format(mean_squared_error(y_test, pred)))\n",
    "print()\n",
    "linreg = LinearRegression()\n",
    "linreg.fit(X_train_extreme, y_train)\n",
    "print('Train set')\n",
    "pred = linreg.predict(X_train_extreme)\n",
    "print('Linear Regression mse: {}'.format(mean_squared_error(y_train, pred)))\n",
    "print('Test set')\n",
    "pred = linreg.predict(X_test_extreme)\n",
    "print('Linear Regression mse: {}'.format(mean_squared_error(y_test, pred)))\n",
    "print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "51127259.0"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round(2212393764.7463093-2161266505.3352327,0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, replacing NA by a value at the end of the distribution, resulted in a lower mean squared error. These means that the difference between the real value and the estimated value is smaller, and thus our model performs better.\n",
    "\n",
    "There is a difference of ~50 million between the model that replaces with the median and the one that uses the far end distribution. So the overall value may seem similar, but when we boil it down to business value, the impact is massive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Ridge Regression mse: 1162112961.4104362\n",
      "Test set\n",
      "Ridge Regression mse: 2203311969.187996\n",
      "\n",
      "Train set\n",
      "Ridge Regression mse: 1216375050.7253299\n",
      "Test set\n",
      "Ridge Regression mse: 2153551540.4403453\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# same as above but using a regularised linear regression\n",
    "linreg = Ridge(random_state=30, max_iter=5, tol=100, alpha=10)\n",
    "linreg.fit(X_train_median, y_train)\n",
    "print('Train set')\n",
    "pred = linreg.predict(X_train_median)\n",
    "print('Ridge Regression mse: {}'.format(mean_squared_error(y_train, pred)))\n",
    "print('Test set')\n",
    "pred = linreg.predict(X_test_median)\n",
    "print('Ridge Regression mse: {}'.format(mean_squared_error(y_test, pred)))\n",
    "print()\n",
    "\n",
    "linreg = Ridge(random_state=30, max_iter=5, tol=100, alpha=10)\n",
    "linreg.fit(X_train_extreme, y_train)\n",
    "print('Train set')\n",
    "pred = linreg.predict(X_train_extreme)\n",
    "print('Ridge Regression mse: {}'.format(mean_squared_error(y_train, pred)))\n",
    "print('Test set')\n",
    "pred = linreg.predict(X_test_extreme)\n",
    "print('Ridge Regression mse: {}'.format(mean_squared_error(y_test, pred)))\n",
    "print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "49760429.0"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round(2203311969.187996-2153551540.4403453,0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same conclusion for regularised linear regression. Go ahead and investigate the impact of this variable replacement on the outliers, if there were any."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**That is all for this demonstration. I hope you enjoyed the notebook, and see you in the next one.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
